common/crypto/keccak: vendor in golang.org/x/crypto/sha3 by Sahil-4555 · Pull Request #18947 · erigontech/erigon

Sahil-4555 · 2026-02-04T01:13:43Z

The upstream golang.org/x/crypto/sha3 package has removed the assembly-optimized Keccak implementation. To avoid a performance regression in Erigon’s hashing hot paths, we maintain a local Keccak implementation that preserves the amd64 assembly code instead of relying on golang.org/x/crypto/sha3.

Refrence: ethereum/go-ethereum#33323
And: golang/go#75486

--

To verify that keccakf.go, keccakf_amd64.go, keccakf_amd64.s, and sha3.go are identical matches to golang.org/x/crypto@v0.43.0, the version right before the asm was removed.

$ gh api "repos/golang/crypto/contents/sha3/keccakf.go?ref=v0.43.0" --jq '.content' | base64 -d > /tmp/upstream_keccakf.go
$ gh api "repos/golang/crypto/contents/sha3/keccakf_amd64.go?ref=v0.43.0" --jq '.content' | base64 -d > /tmp/upstream_keccakf_amd64.go
$ gh api "repos/golang/crypto/contents/sha3/keccakf_amd64.s?ref=v0.43.0" --jq '.content' | base64 -d > /tmp/upstream_keccakf_amd64.s
$ gh api 'repos/golang/crypto/contents/sha3/sha3.go?ref=v0.43.0' --jq '.content' | base64 -d > /tmp/upstream_sha3.go

$ diff -u /tmp/upstream_keccakf_amd64.go crypto/keccak/keccakf_amd64.go
$ diff -u /tmp/upstream_keccakf_amd64.s crypto/keccak/keccakf_amd64.s
$ diff -u /tmp/upstream_legacy_keccakf.go crypto/keccak/keccakf.go
$ diff -u /tmp/upstream_sha3.go crypto/keccak/sha3.go

The hashes.go and sha3_test.go just consists of deletions of unrelated sha3 code.

$ gh api "repos/golang/crypto/contents/sha3/hashes.go?ref=v0.43.0" --jq '.content' | base64 -d > /tmp/upstream_hashes.go
$ gh api "repos/golang/crypto/contents/sha3/sha3_test.go?ref=v0.43.0" --jq '.content' | base64 -d > /tmp/upstream_sha3_test.go

$ diff -u /tmp/upstream_hashes.go crypto/keccak/hashes.go
$ diff -u /tmp/upstream_sha3_test.go crypto/keccak/sha3_test.go

You can check that the asm optimization is still working by running:

go test ./crypto/keccak/... -bench=. -benchmem -count=5 | tee bench_asm.txt
go test -tags=purego ./crypto/keccak/... -bench=. -benchmem -count=5 | tee bench_purego.txt
benchstat bench_purego.txt bench_asm.txt

And all internal uses were successfully ported to our vendored packaged.

$ grep -r 'golang.org/x/crypto/sha3' --include='*.go' .

Sahil-4555 · 2026-02-04T04:01:05Z

@AskAlexSharov
We are still importing golang.org/x/crypto/sha3 in common/crypto/crypto_test.go for TestKeccak256HasherNew. This usage is limited to tests only and is used for output verification, not for production hashing paths. All runtime Keccak usage has been migrated to the local crypto/keccak package. need your feedback before doing any changes

AskAlexSharov · 2026-02-04T07:37:43Z

@Sahil-4555 thank you for picking it up. Will check

yperbasis · 2026-02-11T08:45:14Z

@Sahil-4555 Thank you very much for this work! However, we'll probably go with https://github.com/Giulio2002/fastkeccak. Closing in favour of PR #19092.

yperbasis

keccakf_arm64.s doesn't compile on macos: https://github.com/erigontech/erigon/actions/runs/21910296812/job/63266404162

Sahil-4555 · 2026-02-13T04:36:48Z

@chfast
I’d like to get some feedback on this.

The fastkeccak implementation by Giulio now includes an ARM version which i have added in this PR in addition to the existing AMD64 one. From bechmark observations, Giulio’s implementation appears to have zero allocations, whereas the current version (based on Go’s official crypto/sha3) results in ~32B/op.

In terms of performance, I didn’t observe any consistent or significant speed difference between the two implementations in benchmarks. Based on the feedback here, we can move forward with the preferred approach. If this PR is no longer needed, I’m happy to close it.

Would appreciate your thoughts.

package test

import (
	"fmt"
	"testing"

	keccak "github.com/Giulio2002/fastkeccak"
	"github.com/erigontech/erigon/common/crypto"
)

var benchSizes = []int{32, 128, 256, 1024, 4096, 500 * 1024}

func benchName(size int) string {
	switch {
	case size >= 1024:
		return fmt.Sprintf("%dK", size/1024)
	default:
		return fmt.Sprintf("%dB", size)
	}
}

func BenchmarkCryptoKeccakHasher(b *testing.B) {
	for _, size := range benchSizes {
		data := make([]byte, size)
		for i := range data {
			data[i] = byte(i)
		}
		b.Run(benchName(size), func(b *testing.B) {
			b.SetBytes(int64(size))
			b.ReportAllocs()
			h := crypto.NewKeccakState()
			for b.Loop() {
				h.Reset()
				h.Write(data)
				h.Sum(nil)
			}
		})
	}
}

func BenchmarkFasterKeccakHasher(b *testing.B) {
	for _, size := range benchSizes {
		data := make([]byte, size)
		for i := range data {
			data[i] = byte(i)
		}
		b.Run(benchName(size), func(b *testing.B) {
			b.SetBytes(int64(size))
			b.ReportAllocs()
			var h keccak.Hasher
			for b.Loop() {
				h.Reset()
				h.Write(data)
				h.Sum256()
			}
		})
	}
}

Copilot

Pull request overview

This PR vendors a local common/crypto/keccak implementation (based on golang.org/x/crypto/sha3 prior to the asm removal) and rewires Erigon’s “legacy Keccak” call sites to use it, aiming to avoid hashing hot-path performance regressions.

Changes:

Added a vendored common/crypto/keccak package (including amd64/arm64 optimized permutations and KAT tests).
Replaced golang.org/x/crypto/sha3 legacy Keccak usage with common/crypto/keccak across networking, execution, commitment/trie, DB, and txnprovider code.
Updated benchmarks/tests that instantiate legacy Keccak hashers to use the vendored package.

Reviewed changes

Copilot reviewed 33 out of 36 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
txnprovider/txpool/pool_txn_parser.go	Switch txn parsing contexts to vendored legacy Keccak instances.
txnprovider/shutter/internal/crypto/hash.go	Switch shutter keccak256 helper to vendored legacy Keccak.
polygon/bor/bor.go	Use vendored legacy Keccak for merkle tree hashing.
p2p/rlpx/rlpx.go	Use vendored legacy Keccak for RLPx MAC setup.
p2p/enode/idscheme.go	Use vendored legacy Keccak for ENR signing/verification hashes.
p2p/dnsdisc/tree.go	Use vendored legacy Keccak for DNS discovery entry hashing/signature hashes.
node/shards/state_cache_test.go	Switch test hasher construction to vendored legacy Keccak.
execution/vm/instructions.go	Use vendored legacy Keccak for EVM `KECCAK256` opcode hasher initialization.
execution/tests/testutil/state_test_util.go	Use vendored legacy Keccak for RLP hashing in test utilities.
execution/rlp/rlp_test.go	Use vendored legacy Keccak for hashing benchmarks.
execution/protocol/rules/ethash/rules.go	Use vendored legacy Keccak for Ethash seal hash.
execution/protocol/rules/ethash/algorithm.go	Use vendored legacy Keccak{256,512} for Ethash seed/cache/dataset hashing.
execution/protocol/block_exec.go	Use vendored legacy Keccak for RLP hash during block execution.
execution/protocol/aa/validation_rules_tracer.go	Use vendored legacy Keccak for associated storage hashing in tracer logic.
execution/commitment/trie/hasher.go	Update hasher pool to vendored legacy Keccak while keeping `crypto.KeccakState` assertions.
execution/commitment/trie/hashbuilder.go	Update `HashBuilder` to vendored legacy Keccak state.
execution/commitment/trie/account_node_test.go	Update test hashing to vendored legacy Keccak state.
execution/commitment/patricia_state_mock_test.go	Update mock state hashing to vendored legacy Keccak.
execution/commitment/hex_patricia_hashed.go	Use vendored legacy Keccak for Patricia-hashed trie internals.
execution/commitment/commitment.go	Use vendored legacy Keccak in commitment validation path.
db/rawdb/accessors_chain_test.go	Use vendored legacy Keccak in rawdb storage tests.
db/kv/kvcache/cache.go	Use vendored legacy Keccak in cache hasher initialization.
common/hasher.go	Switch common hasher pool to vendored legacy Keccak.
common/crypto/keccak/sha3_test.go	Adds upstream-derived KATs/benchmarks for the vendored Keccak implementation.
common/crypto/keccak/sha3.go	Adds upstream-derived sponge/hash implementation used by legacy Keccak.
common/crypto/keccak/keccakf_arm64.go	Declares arm64 permutation hook (asm-backed) under build tags.
common/crypto/keccak/keccakf_amd64.s	Adds amd64 assembly permutation for keccak-f1600.
common/crypto/keccak/keccakf_amd64.go	Declares amd64 permutation hook (asm-backed) under build tags.
common/crypto/keccak/keccakf.go	Adds pure-Go permutation for non-asm builds/architectures.
common/crypto/keccak/hashes.go	Adds constructors for legacy Keccak-256/512.
common/crypto/keccak/README.md	Documents why the vendored Keccak exists and why stdlib SHA-3 can’t be used.
common/crypto/keccak/LICENSE	Adds upstream license for vendored code.
common/crypto/crypto.go	Switch core crypto helpers/pools to vendored legacy Keccak.
common/address.go	Switch checksum hashing to vendored legacy Keccak.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-17T09:02:38Z

node/shards/state_cache_test.go

 	sc.SetStorageWrite(a2.Bytes(), 1, l3.Bytes(), nil)
 	lastK = lastK[:0]
-	if err := sc.WalkStorage(common.BytesToHash(sha3.NewLegacyKeccak256().Sum(a1.Bytes())), 1, nil, func(locHash common.Hash, val []byte) error {
+	if err := sc.WalkStorage(common.BytesToHash(keccak.NewLegacyKeccak256().Sum(a1.Bytes())), 1, nil, func(locHash common.Hash, val []byte) error {


hash.Hash.Sum does not hash the provided slice; it appends the current digest to the argument. As written, this computes a1.Bytes() || keccak256(empty) instead of keccak256(a1.Bytes()), so WalkStorage will likely not visit the intended entries and the test won't actually validate ordering. Compute the address hash by writing a1.Bytes() into the hasher (or use an existing helper like crypto.Keccak256Hash).

Suggested change

if err := sc.WalkStorage(common.BytesToHash(keccak.NewLegacyKeccak256().Sum(a1.Bytes())), 1, nil, func(locHash common.Hash, val []byte) error {

hasher := keccak.NewLegacyKeccak256()

if _, err := hasher.Write(a1.Bytes()); err != nil {

t.Fatalf("failed to hash address: %v", err)

}

addrHash := common.BytesToHash(hasher.Sum(nil))

if err := sc.WalkStorage(addrHash, 1, nil, func(locHash common.Hash, val []byte) error {

Giulio2002 · 2026-02-20T17:35:30Z

closing in favor of mine

Sahil-4555 requested review from AskAlexSharov, Giulio2002, mh0lt, sudeepdino008, taratorio and yperbasis as code owners February 4, 2026 01:13

AskAlexSharov mentioned this pull request Feb 4, 2026

ARM64: Trie computation performance during sync/recovery #18398

Open

yperbasis added the performance label Feb 8, 2026

AskAlexSharov mentioned this pull request Feb 11, 2026

Performance: use NEON accelerated keccak #19092

Merged

yperbasis closed this Feb 11, 2026

yperbasis reopened this Feb 11, 2026

Sahil-4555 requested review from bloxster, canepat and domiwei as code owners February 11, 2026 14:58

Sahil-4555 force-pushed the crypto-keccak-vedoring branch 2 times, most recently from ac7bb03 to 9d0cd99 Compare February 11, 2026 15:02

yperbasis requested changes Feb 11, 2026

View reviewed changes

Sahil-4555 force-pushed the crypto-keccak-vedoring branch from 9d0cd99 to eced6d7 Compare February 11, 2026 16:15

Sahil-4555 added 6 commits February 11, 2026 21:47

common/crypto/keccak: vendor in golang.org/x/crypto/sha3

dbcd8f8

all: use vendored keccak package

9bc6be2

all: use vendored keccak package

5e78e5f

optimize version

919ab33

common/crypto/keccak: vendor in golang.org/x/crypto/sha3

73425f6

fix CLI issue

dec569b

Sahil-4555 force-pushed the crypto-keccak-vedoring branch from eced6d7 to dec569b Compare February 12, 2026 05:48

Sahil-4555 requested a review from yperbasis February 12, 2026 09:21

Merge branch 'main' into crypto-keccak-vedoring

de3fe48

Sahil-4555 requested review from anacrolix, antonis19 and awskii as code owners February 13, 2026 12:08

Sahil-4555 and others added 2 commits February 13, 2026 17:40

resolve conflicts

d023ca0

Merge branch 'main' into crypto-keccak-vedoring

8d75852

yperbasis self-assigned this Feb 14, 2026

fix lint

74b5b43

Sahil-4555 force-pushed the crypto-keccak-vedoring branch from 4f186a1 to 74b5b43 Compare February 16, 2026 09:56

Sahil-4555 and others added 2 commits February 16, 2026 15:27

Merge branch 'main' into crypto-keccak-vedoring

7049633

Merge branch 'main' into crypto-keccak-vedoring

37915b1

yperbasis requested a review from Copilot February 17, 2026 08:58

Copilot started reviewing on behalf of yperbasis February 17, 2026 08:59 View session

Copilot AI reviewed Feb 17, 2026

View reviewed changes

Giulio2002 closed this Feb 20, 2026

Sahil-4555 deleted the crypto-keccak-vedoring branch March 10, 2026 12:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

common/crypto/keccak: vendor in golang.org/x/crypto/sha3#18947

common/crypto/keccak: vendor in golang.org/x/crypto/sha3#18947
Sahil-4555 wants to merge 12 commits intoerigontech:mainfrom
Sahil-4555:crypto-keccak-vedoring

Sahil-4555 commented Feb 4, 2026 •

edited by AskAlexSharov

Loading

Uh oh!

Sahil-4555 commented Feb 4, 2026

Uh oh!

AskAlexSharov commented Feb 4, 2026

Uh oh!

yperbasis commented Feb 11, 2026

Uh oh!

yperbasis left a comment

Uh oh!

Sahil-4555 commented Feb 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 17, 2026

Uh oh!

Giulio2002 commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

-	if err := sc.WalkStorage(common.BytesToHash(keccak.NewLegacyKeccak256().Sum(a1.Bytes())), 1, nil, func(locHash common.Hash, val []byte) error {
+	hasher := keccak.NewLegacyKeccak256()
+	if _, err := hasher.Write(a1.Bytes()); err != nil {
+		t.Fatalf("failed to hash address: %v", err)
+	}
+	addrHash := common.BytesToHash(hasher.Sum(nil))
+	if err := sc.WalkStorage(addrHash, 1, nil, func(locHash common.Hash, val []byte) error {

Conversation

Sahil-4555 commented Feb 4, 2026 • edited by AskAlexSharov Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sahil-4555 commented Feb 4, 2026

Uh oh!

AskAlexSharov commented Feb 4, 2026

Uh oh!

yperbasis commented Feb 11, 2026

Uh oh!

yperbasis left a comment

Choose a reason for hiding this comment

Uh oh!

Sahil-4555 commented Feb 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

Giulio2002 commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Sahil-4555 commented Feb 4, 2026 •

edited by AskAlexSharov

Loading